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ABSTRACT 

The  use  of  Combined  Arms,  a  very  important  set  of  tactics,  techniques,  and  procedures  to  the  Marine  Corps,  is 
complex  and  requires  considerable  practice  and  training  of  the  entire  Combined  Arms  (CA)  team.  Simulations  exist 
and  are  being  developed  that  provide  a  practice  and  training  environment  for  the  Combined  Arms  team.  However, 
for  effective  training  to  occur,  use  of  simulations  requires  intelligent  evaluation  and  feedback.  Furthermore,  as 
practice  develops  the  skills  of  the  team  and  prepares  it  to  handle  more  difficult  scenarios,  additional  training 
scenarios  should  become  more  and  more  complex  and  realistic.  A  CA  training  opportunity  was  identified  and 
addressed  through  the  development  of  an  Intelligent  Tutoring  System  (ITS).  The  issue  was  that  no  simulation 
existed  which  provided  intelligent  feedback  to  the  Fire  Support  Team  (FiST)  as  to  the  correctness  of  their  CA  plan. 
This  prevented  them  from  receiving  enough  practice  and  training  in  important  aspects  of  this  planning  process 
which  impacted  the  quality  of  Combined  Arms  Exercise  (CAX)  training  opportunities.  Therefore  an  ITS  which 
could  evaluate  and  provide  feedback  on  the  FiST  developed  CA  plan  was  developed.  This  paper  first  covers  the 
training  needs  for  CA  and  the  FiST  then  discusses  ITSs  generally  and  how  their  capabilities  meet  the  identified 
needs.  The  development  methodology  used  for  this  project  is  described.  The  functional  capabilities  of  the  ITS  are 
discussed  followed  by  the  underlying  architecture  that  implemented  those  capabilities.  The  paper  finishes  with 
results,  lessons  learned,  and  future  work. 
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COMBINED  ARMS  TEAM  AND  FiST  TRAINING 
NEEDS 

The  Marine  Corps  places  strong  emphasis  on  the  use  of 
Combined  Arms  in  battle.  Combined  Arms  is  the  art 
of  combining  the  effects  of  mortars,  artillery,  naval 
gunfire,  close  air  support,  and  close-in  fire  support  to 
inflict  the  maximum  amount  of  damage  on  the  enemy 
in  a  minimum  amount  of  time  to  support  a  maneuver 
element's  assault  on  the  enemy  [MCAGCC  2000]. 
However,  this  is  a  complex  effort  that  requires  the 
involvement  of  hundreds  of  warfighters  with  dozens  of 
different  specialties.  An  important  component  of  the 
combined  arms  team  is  the  Fire  Support  Team  (FiST). 
The  FiST  includes  the  FiST  Leader,  Forward  Air 
Controller,  Artillery  Forward  Observer,  Mortar 
Forward  Observer,  and  Naval  Gun  Fire  Spotter. 
Assistants  and  radio  operators  complement  each  of 
these.  The  FiST's  tasks,  like  that  of  many  tactical 
decision-makers  divide  themselves  into  two  categories 
-  those  that  relate  to  planning  and  those  that  relate  to 
mission  execution.  A  critically  important  planning  task 
of  the  FiST  is  developing  the  battle  space  geometry 
(BSG).  The  BSG  is  a  representation  of  the  effects  and 
possible  effects  (such  as  minimum  safe  distances)  of 
the  different  types  of  supporting  fire.  These  must  be 
integrated  and  deconflicted  with  each  other  and  the 
maneuver  units.  Other  considerations  and  principles 
also  apply,  such  as  the  importance  of  using  all 
available  assets,  redundancy,  and  other  concepts 
typical  of  ground  unit  tactical  planning.  Once  the  FiST 
has  developed  a  plan,  approval  for  it  is  requested  up 
the  chain  of  command  before  execution  may 
commence.  Once  execution  begins,  planning  and 
deconfliction  of  fires  is  continual. 

The  complexity  of  Combined  Arms  (CA)  and  the  large 
number  of  members  of  the  extended  CA  team  dictate 
that  considerable  practice  and  training  is  required.  An 
important  component  of  CA  training  is  the  CA 
Exercise  (CAX).  CAX  instructors  have  identified  a 
training  deficit  -  FiSTs  arriving  at  CAX  are  not 
familiar  enough  or  had  enough  practice  with  BSG. 

This  is  because  there  is  no  existing  means  like  a 
training  simulator  that  could  provides  intelligent 
evaluation  and  feedback  for  BSG.  The  effort  described 
in  this  paper  directly  addresses  this  identified  deficit. 


Nancy  Harmon 

MARCORSYSCOM,  PMTRASYS 
Orlando,  FL 

nancy.harmon@navy.mil 

This  is  an  especially  important  training  gap  to  close, 
given  the  relationship  between  BSG  and  safety. 
According  to  the  experts  and  instructors,  the  most 
important  parameter  in  developing  effective  BSG 
knowledge  and  skills  is  practice. 

CAX  training  consists  of  both  live  exercises  and 
simulation  training.  The  Combined  Arms  Staff  Trainer 
(CAST)  has  been  in  use  for  many  years  at  several  sites 
for  the  purpose  of  CA  team  training.  A  current 
development  effort,  the  CAST  Upgrade,  seeks  to 
implement  improvements  to  CAST  to  further  improve 
its  simulated  scenario  team  training  capabilities.  The 
CAST  Upgrade  is  itself  a  precursor  to  the  Combined 
Arms  Command  &  Control  Tactical  Upgrade  System 
(CACCTUS)  project.  The  effort  described  here  is  part 
of  the  CAST  Upgrade  which  is  developing  a 
simulation  tool  to  provide  an  automatic  debriefing  of 
the  BSG  created  by  the  FiST  in  a  simulated  tactical 
scenario.  This  tool  would  be  invaluable  for  every 
captain  going  to  the  Expeditionary  Warfare  School  to 
receive  BSG  training  though  the  use  of  this  tool.  The 
goal  of  the  CAST  Upgrade  is  to  integrate  the  ITS  with 
existing  tools  and  training  simulation  like  that  of 
OneSAF  to  provide  practice  of  the  BSG  techniques  and 
the  ability  for  mission  rehearsal  planning.  The  tool 
must  evaluate  the  FiST's  BSG  and  make  sure  the 
trainees  realize  the  possible  ramifications  of  a  mistake. 
It  should  gradually  increase  the  complexity  of 
scenarios  as  the  particular  FiST  demonstrates  that  they 
are  ready  for  this  increased  complexity.  This  requires 
that  the  software  infer  the  state  of  the  team's 
knowledge  and  its  ability  to  practically  apply  that 
knowledge  in  an  operational  context.  It  should  retest 
the  FiST  in  appropriate  scenarios  to  determine  the 
effectiveness  of  the  debriefing  and  to  ensure  the 
trainees  have  mastered  the  required  knowledge  and 
skills. 

For  maximum  flexibility,  many  scenarios  should  be 
developed.  The  Marine  Corps  therefore  needs  to  have 
the  flexibility  to  create  and  customize  their  own 
scenarios  in  a  user-friendly  interface.  By  allowing 
Marine  or  instructor  creation/modification  of  tactical 
scenarios,  turn-around  time  will  be  greatly  reduced  and 
precious  development  dollars  saved. 
Miscommunication  and  communication  overhead 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2003 


between  instructor  and  developer  is  eliminated  since 


Figure  1.  ITS 

INTELLIGENT  TUTORING  SYSTEM  (ITS) 

CAPABILITIES  MEETING  THOSE  NEEDS 

To  increase  BSG  scenario  practice,  as  recommended 
above,  will  require  that  the  evaluation  and  feedback  of 
BSG  be  performed  by  software.  Automatic  evaluation 
of  decisions  in  simulated  scenarios  is  an  important 
component  of  the  field  of  Intelligent  Tutoring  Systems 
(ITSs).  The  ITS  should  work  in  concert  with  a 
common  PC  planning  tool  and  training  simulation,  to 
facilitate  increased  access.  ITSs  automatically  monitor 
and  assess  trainee  performance  in  simulated  tactical 
scenarios  and  evaluate  the  trainee's  mastery  and 
progress.  This  information  is  used  for  debriefing  after 
the  performance  is  complete.  In  the  case  of  BSG,  this 
corresponds  to  after  the  CA  plan,  including  the  BSG, 
has  been  entered.  Associated  with  the  debriefing  may 
also  be  of  appropriate  multimedia  descriptive  material. 
The  debriefing  is  one  form  of  remediation.  Others  are 
presentation  of  related  examples  (often  in  the  training 
simulator  itself)  and  further  practice  in  carefully 
chosen  scenarios.  Typically  the  trainee  would  be  re¬ 
assessed  in  simulated  scenarios  to  determine  the 
effectiveness  of  the  remediation.  This  ability  to  adapt 
and  customize  itself  to  the  trainee  requires  that  the  ITS 
have  a  well-developed  model  of  the  student. 


typical  ITS  -  a  Student  Assessment  Module,  a  Student 
Model,  and  an  Instructional  Planner.  The  Student 
Assessment  Module  takes  as  input  student  observables 
from  a  simulation  or  other  application,  considers 
background  information,  then  determines  the 
correctness  of  the  student's  decisions  and  updates  the 
Student  Model.  An  important  point  is  that  the  ITS  can 
only  base  its  assessment  of  the  trainee  on  what  it  can 
actually  observe  electronically.  This  requires  that  all 
relevant  trainee  inputs  be  digital  in  nature  and  that  the 
ITS  has  access  to  them  which  usually  entails  an 
interface  to  the  training  simulation  (considered  part  of 
the  user  interface  in  this  figure).  The  Battlespace 
Geometry  Tutor  (BSGT)  concentrates  primarily  on  the 
assessment  module  but  includes  aspects  of  the  other 
components  as  well.  The  Student  Model  may  include 
estimates  of  many  short  and  long-term  student 
attributes  but  should  minimally  include  estimates  of  the 
mastery  of  relevant  knowledge  and  skills.  The 
Instructional  Planner  makes  instructional  decisions 
including  the  type  and  timing  of  feedback  as  well  as 
what  scenarios  would  be  most  appropriate  for  practice 
or  illustration. 


BSG  ITS  DESCRIPTION 
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Planning  Phase  Tutoring 

As  alluded  to  earlier,  the  FiST  performs  during  two 
distinct  mission  phases  -  planning  and  execution. 
(Although  planning  and  deconfliction  continue 
throughout  execution.)  Consequently,  FiST  training 
consists  of  two  corresponding  phases  as  well.  Our 
tutoring  system  concentrates  on  the  first  phase  -  initial 
fires  planning,  specifically  the  creation,  integration, 
and  deconfliction  of  BSG.  The  BSG  ITS,  called  the 
BSG  Tutor  (BSGT),  first  presents  a  scenario  briefing 
to  the  FiST,  which  describes  the  tactical  situation  and 
additional  instructions.  Normally  the  situation  given  to 
the  FiST  includes  the  unit  commander's  scheme  of 
maneuver  and  often  the  timing  and  location  of 
proposed  aircraft  delivered  munitions.  The  trainees 
then  view  the  tactical  scenario  in  C2PC,  the  Marine 
Corps  Command  and  Control  software  for  transmitting 
or  receiving  the  Common  Tactical  Picture  (CTP)  or 
Common  Operational  Picture  (COP)  in  a  Windows 
environment.  Its  main  display  is  a  2-D  tactical  map. 
Most  of  the  FiST's  time  with  BSGT  is  actually  spent 
within  C2PC.  Using  a  specialized  C2PC  add-on 
developed  by  MTS,  the  FiST  creates  a  CA  quickfire 
plan.  (A  quickfire  plan  is  one  developed  rapidly, 
usually  when  contact  is  first  made  with  an  enemy  unit). 
The  plan  consists  of  the  timing  and  location  of 
suppressive  fires,  marking  rounds,  aircraft  delivered 
ordinance,  rotary  wing  battle  positions  and  routes,  and 
fixed  wing  attack  cones  and  routes.  Associated  with 
each  of  these  are  various  three  dimensional  shapes  that 
describe  minimum  safe  distances,  exclusion  and  safety 
zones,  and  the  time  periods  when  each  of  these  are 
active.  When  the  team  believes  it  has  completed  its 
plan,  it  requests  an  evaluation.  The  BSGT  first 
automatically  checks  that  they  have  defined  the  correct 
shapes  with  the  correct  parameters  associated  with 
each  munition  delivery.  For  example,  the  FiST  should 
have  shown  a  400  meter  radius  circle  around  a  mortar 
target,  corresponding  to  its  minimum  safe  distance  and 
a  rectangle  whose  centerline  is  the  gun  target  line 
(GTL)  from  the  firing  mortar  unit  to  the  targeted 
enemy  unit  and  whose  width  is  800  meters.  They 
should  have  also  drawn  routes  between  the  various 
battle  positions.  If  any  of  these  are  omitted  or 
incorrect,  the  debriefing  informs  him. 

The  real  heart  of  the  automatic  plan  evaluation  is 
checking  whether  the  various  aspects  of  the  plan  have 
any  conflicts.  This  checking  occurs  immediately  after 
the  correct  shape  checking  described  above.  Since 
there  are  four  main  categories  of  the  plan  (maneuver 
actions,  indirect  fires,  fixed  wing  delivered  munitions, 
and  rotary  wing  delivered  munitions)  there  are  6  main 
categories  of  possible  conflicts  corresponding  to  all  of 


the  possible  pairs  of  the  four  main  plan  element 
categories.  These  are  conflicts  between  maneuver  and 
indirect  fire;  between  maneuver  and  fixed  wing 
operations;  maneuver  and  rotary  wing  operations; 
indirect  fire  and  fixed  wing  operations;  indirect  fire 
and  rotary  wing  operations;  and  between  fixed  wing 
operations  and  rotary  wing  operations.  Checking  for  a 
conflict  between  any  pair  of  plan  elements  entails  first 
checking  that  they  are  both  active  during  some 
overlapping  period  of  time  and  that  the  shape 
associated  with  each  overlaps  in  three  dimensional 
space.  A  small  number  of  checks,  out  of  the  dozens 
that  BSGT  performs,  are  mentioned  below,  as 
examples.  BSGT  checks  if  a  maneuver  element  such 
as  a  support  by  fire  position  or  axis  of  attack  is  within 
the  minimum  safe  distance  circle  of  fixed  wing,  rotary 
wing,  mortar,  and  artillery  munitions  when  both  are 
active.  It  checks  that  the  fixed  wing  attack  cone  is  at 
least  1000  vertical  feet  from  the  trajectory  of  artillery 
where  the  GTL  crosses  the  attack  cone.  A  final 
example  is  that  it  checks  that  a  rotary  wing  battle 
position  is  not  inside  the  threat  ring  of  an  unsuppressed 
enemy  air  defense  unit. 

Most  of  feedback  on  discovered  conflicts  may  be  given 
immediately.  This  feedback  is  usually  in  the  form  of 
text  describing  the  time,  location  and  which  aspects  of 
the  plan  are  in  conflict.  The  system  can  also  show  the 
three  dimensional  shapes  that  are  in  conflict  in  a  three 
dimensional  terrain  viewer,  upon  trainee  request.  In 
cases  where  the  conflict  relates  primarily  to  timing, 
BSGT  can  call  OTB  (the  OneSAF  Test  Bed),  skip  it 
ahead  to  the  time  of  the  conflict  and  show  the  conflict 
occurring  at  that  time.  For  example,  if  the  conflict  is 
the  result  of  a  planned  maneuver  of  dismounted 
infantry  toward  the  objective  at  a  certain  time  and  the 
team  has  failed  to  halt  the  artillery  suppressive  fire  on 
the  same  target  early  enough,  then  BSGT  would  call 
OTB  with  the  scenario  and  skip  the  scenario  ahead  to 
the  time  where  the  dismounted  infantry  is  entering  the 
minimum  safe  distance  circle  for  that  artillery.  It  can 
then  show  the  dismounted  infantry  unit  destroyed  by 
that  friendly  artillery.  In  the  case  of  a  continually 
repeated  mistake,  perhaps  because  the  trainees  are 
relying  too  heavily  on  the  conflict  checking 
mechanism,  feedback  on  the  conflict  can  be  delayed 
until  during  the  mission  execution  phase,  where  having 
the  friendly  infantry  unit  destroyed  by  friendly  artillery 
is  more  grievous. 

Since  the  conflict  checker  of  BSGT  included 
capabilities  for  checking  conflicts  between  shapes,  line 
segments,  curves,  and  points,  it  was  also  useful  for 
checking  for  unplanned  conflicts  during  the  real-time 
simulation.  This  was  also  deemed  useful  for  the  larger 
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CAST  UP  system.  Therefore  it  was  modularized  and 
provided  as  a  general  network  resource  that  constantly 
monitored  the  activities  of  various  plan  entities  and 
broadcast  any  discovered  conflicts.  This  information 
could  be  used  by  an  instructor  monitoring  the  exercise 
or  a  different,  future  ITS  which  evaluated  real-time 
tactical  decision-making. 

The  FiST's  knowledge  of  other  principles  relating  to 
developing  the  combined  arms  plan  is  also  evaluated. 
Examples  of  these  are  making  sure  marking  rounds  are 
planned  for  fixed  wing  munitions,  that  the  battlefield 
will  be  clean  for  rotary  wing  delivered  laser  guided 
munitions  such  as  the  Hellfire  missile,  and  both 
mortars  and  artillery  are  employed  whenever  possible. 
This  feedback  is  generally  given  immediately. 

As  an  Intelligent  Tutoring  System,  BSGT  provides 
other  instruction  besides  feedback  on  the  correctness  of 
the  team’s  CA  plan.  One  of  the  more  important 
functions,  outside  of  evaluation  is  determining  when 
the  FiST  is  ready  for  more  complicated,  more 
challenging  scenarios.  It  makes  this  decision  based  on 
its  estimate  of  the  team's  mastery  of  the  techniques, 
procedures,  and  knowledge  required  for  CA  planning. 
This  multi-dimensional  collection  of  estimates  is  the 
student  model.  Each  element  of  the  plan  and  each 
possible  kind  of  conflict  has  several  principles 
associated  with  it.  Whenever  a  component  of  the 
team's  plan  is  correct,  BSGT  increases  its  estimate  of 
the  team's  mastery  of  the  associated  principles  and 
whenever  a  component  is  incorrect,  it  decreases  its 
estimate.  Initially  the  trainees  perform  the  simplest 
scenarios.  When  the  large  majority  of  mastery 
estimates  reach  the  intermediate  level,  more  complex 
scenarios  are  selected  for  the  team.  Additionally 
within  a  single  level  of  complexity,  scenarios  will  be 
chosen  that  include  components  with  associated 
principles  that  the  FiST  has  not  been  tested  on  or  has 
done  poorly  with  (to  practice  its  weakest  areas).  The 
team  can  examine  these  mastery  estimates  by  looking 
at  BSGT's  student  model  for  them.  Because  trainees 
using  BSGT  have  already  received  instruction  on  BSG, 
BSGT  assumes  that  they  are  at  least  ready  to  attempt 
the  simplest  scenarios  available  to  it  without  additional 
instruction  so  BSGT  does  not  include  mechanisms  for 
introducing  the  concepts  to  beginners. 

Use  of  BSGT  Capabilities  During  Real-Time 
Mission  Execution 

As  mentioned  above,  during  mission  execution, 
conflict  checking  continues  to  be  active.  Found 
conflicts  are  provided  to  the  instructor  immediately. 
Options  that  the  instructor  has  include  3-D  BSG 


conflict  shape  visualization  or  immediately  destroying 
the  friendly  unit  involved  in  the  conflict  or  simply 
pointing  out  the  problem  to  the  FiST. 

An  additional  capability  from  BSGT  was  considered 
potentially  useful  for  the  instructors.  Since  the  C2PC 
symbology  had  to  be  interpreted  to  be  evaluated  during 
the  planning  stage  anyway,  it  is  possible  that  this 
capability  could  be  provided  during  real-time 
execution  to  create  a  user-friendly  way  of  controlling 
the  enemy  and  friendly  units  in  the  scenario.  So,  for 
example,  if  a  ground  unit  movement  arrow  is  shown  in 
the  scenario  plan  for  a  specific  time  period,  an 
automated  controller  could  send  orders  to  OTB  to 
move  the  associated  unit  along  the  path  described  by 
the  arrow  at  the  appropriate  time.  Similarly  if  the 
arrow  ends  in  a  SBF  position,  when  the  unit  reaches  it, 
it  could  be  instructed  to  occupy  a  SBF  position  at  that 
location  oriented  toward  the  target.  It  could  then  be 
instructed  to  fire  during  the  time  periods  described  in 
the  CA  plan.  For  this  level  of  FiST  training,  enemy 
units  aren't  required  to  behave  very  realistically. 
However  the  capability  to  define  sophisticated 
behaviors  for  enemy  units  could  also  be  incorporated. 


Instructor  Authoring 

As  described  in  the  Needs  Section,  the  Marine  Corps 
instructors  need  to  have  the  ability  to  author  scenario 
themselves.  Since  the  instructors  are  already  familiar 
with  C2PC,  it  was  logical  to  use  it  as  a  basis  for  a 
scenario  editor.  Using  the  existing  C2PC,  instructors 
can  create  scenarios  for  both  the  planning  and 
execution  phases  by  placing  friendly  and  enemy  units 
and  symbols  such  as  arrows  and  SBF  positions  that 
show  the  commander's  scheme  of  maneuver.  Using  a 
specialized  C2PC  add-on  developed  by  MTS  for  the 
instructors,  they  can  also  create  a  briefing  that  is 
presented  to  the  FiST  at  the  beginning  of  the  planning 
session.  This  briefing  typically  includes  some  aircraft 
delivered  munitions  and  their  timing.  This  add-on  also 
allows  the  instructors  to  enter  a  correct  CA  plan  with 
associated  BSG  using  a  lot  of  the  same  functionality 
provided  to  the  trainees  with  some  important 
differences.  Unlike  the  trainee  version,  the  instructor 
version  automatically  selects  the  right  parameters 
based  on  the  chosen  munitions.  Additionally,  the 
instructor  can  check  for  conflicts  at  any  time  and  will 
always  get  the  complete  set  of  conflicts  found.  Of 
course,  no  modeling  of  the  instructor's  expertise  occurs 
during  their  editing  process.  Though  not  used  as 
frequently,  the  instructors  can  also  modify  the 
evaluation  process  and  instructional  methods.  The 
evaluations  are  represented  as  Behavior  Transition 
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Networks  (BTNs)  which  are  generalizations  of  finite 
state  machines.  These  are  described  more  fully  in 
[Stottler  &  Jensen  2002]. 

BSGT  Architecture 

The  BSGT  context  is  shown  below.  The  ITS 
Authoring  suite  of  tools,  shown  on  the  right,  is  used  by 
the  subject  matter  expert  (SME)  to  create  a  series  of 
files,  shown  as  ovals  that  together  with  the  domain 
independent  ITS  software  (not  shown)  constitute  the 


BSGT.  On  the  left  is  the  suite  of  applications  that  the 
trainees  interact  with.  As  described  previously  these 
include  using  C2PC  to  initially  view  the  scenario  then 
using  it  with  its  add-ons  to  input  a  CA  plan.  This  plan 
is  received  by  the  ITS  which  evaluates  it  and  presents 
feedback  to  the  trainees  as  shown.  The  feedback  may 
include  execution  of  the  plan  in  the  simulation, 
especially  to  illustrate  timing  problems.  Included  in 
the  simulation  module  of  the  figure  is  a  3-D  viewer  and 
OTB  (later  OOS  (OneSAF  Objective  System). 


Figure  2.  Battlespace  Geometry  Tutor  System  Context 


The  architecture  of  this  Intelligent  Tutoring  System 
(ITS)  is  shown  below  in  figure  3  where  the  ovals  (the 
ITS  files)  have  been  recreated  from  the  previous 
figure.  An  additional  oval,  the  student  model,  is  also 
shown.  The  ITS  Runtime  Engine  consults  the 
instruction  strategy  knowledge  and  the  student  model 
and  decides  on  the  next  instructional  event.  Usually 
that  is  a  planning  scenario  exercise. 

Other  instructional  techniques  of  the  ITS  include 
selecting  scenarios  to  test  untested  principles  or 
exercise  known  trainee  weaknesses.  Scenarios  may 
also  be  selected  and  shown  to  the  team  as  examples  for 
illustrative  purposes.  The  instruction  strategy 
knowledge  also  specifies  when  to  show  principle 
descriptions.  Its  decisions  are  based  on  the  FiST's 
level  of  mastery,  number  of  past  failures,  and 


instruction  already  received.  It  will  generally  present 
easier,  less  complex,  less  workload  intensive  scenarios 
initially  and  increase  these  dimensions  as  the  team 
shows  mastery. 

Actions  performed  during  mission  planning  are  both 
evaluated  by  comparing  them  to  stored  annotated  plans 
and  via  BTNs.  The  results  of  these  evaluations  go 
directly  to  the  ITS  Runtime  Engine  and  are  processed 
and  forwarded  to  the  student  modeling  component. 
The  student  modeling  component  examines  a  team's 
entire  history  with  regard  to  a  specific  skill  or 
knowledge  item  and  estimates  the  FiST’s  mastery  of  it. 
It  does  this  for  all  skills  and  knowledge  items  tested  in 
a  scenario.  The  ITS  Runtime  Engine  forwards  the 
actions  and  principles  that  need  to  be  remediated  to  the 
User  Interface  UI). 
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Figure  3.  BSGT  Architecture 


TASK  ANALYSES,  DESIGN,  DEVELOPMENT, 
AND  EVALUATION  METHODOLOGY 

To  develop  a  beneficial  ITS  requires  following  the 
proper  development  process,  and  ITSs,  like  most 
training  systems  do  actually  follow  the  standard 
ISD's  steps  of  Analyze,  Design,  Develop,  Implement, 
and  Evaluate,  although  the  steps  sometimes  differ  in 
emphasis  and  are  therefore  often  renamed  slightly. 
The  effort  to  develop  the  ITS  for  BSG  also  followed 
a  fairly  standard  ISD  development  methodology. 
The  first  step  is  normally  called  Knowledge 
Engineering  or  Cognitive  Task  Analysis  (CTA).  The 
difference  is  that  there  are  usually  two  parallel  tracks, 
one  for  the  domain  to  be  trained  (in  this  case  the 
planning  process  of  creating,  integrating  and 
deconflicting  the  proper  BSG)  and  one  for 
instructional  methods,  which  are  tasks  performed  by 
the  instructor  that  they  have  found  beneficial  in 
training  these  types  of  trainees  for  these  types  of 
tasks.  For  example,  one  technique  used  by 
instructors  teaching  BSG  is  to  kill  units  manually  that 
have  violated  safety  related  BSG,  even  though  they 
would  have  very  probably  survived  to  illustrate  the 
importance  of  a  BSG  safety  issue.  The  reason  for 


this  second  track  of  CTA  is  that  the  ITS  will  be 
designed  to  support  many  of  these  successful 
instructional  techniques  and  mimic  methods  of 
evaluation  and  feedback.  For  the  BSG  domain,  CTA 
first  entailed  examining  the  relevant  documents. 
[MCAGCC  2000]  turned  out  to  be  a  very  important 
source  of  domain  knowledge.  We  then  went  through 
hypothetical  and  training  scenarios  with  experts, 
noting  their  decisions  and  actions,  and  questioned 
them  in  detail  regarding  each  decision.  We  also 
observed  instructors  in  action  and  discussed  with 
them  various  instructional  techniques,  actions,  and 
decisions.  In  particular  the  instructors  were  able  to 
describe  the  required  scenarios  and  how  they  should 
increase  in  complexity.  We  also  questioned  the 
instructors  as  to  the  most  important  aspects  to  be 
taught  and  common  trainee  errors.  In  general  we 
also  always  stay  alert  for  techniques  for 
automatically  evaluating  the  correctness  of  trainee 
actions. 

After  the  twin  CTA  described  above,  we  developed 
the  design.  Typically  the  design  is  not  simply  based 
on  the  requirements  for  the  best  instructional  ITS 
possible,  because  there  are  always  additional 
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constraints.  A  seemingly  large  constraint  was  that 
the  ITS  would  have  to  be  integrated  into  the  CAST 
Upgrade's  evolving  architecture  and  use  its  existing 
simulation.  However  (and  unusually),  we  did  not 
need  to  make  the  usual  instructional  compromises 
related  to  utilizing  an  existing  system.  This  was 
partly  because  many  of  the  CAST  UP  design 
decisions  were  delayed  until  the  ITS  project  was 
started.  But  it  was  probably  more  related  to  the  fact 
that  the  CAST  UP  architecture  had  already  been 
designed  with  close  collaboration  with  instructors  to 
make  sure  that  they  could  perform  their  duties  well 
and  with  all  the  instructional  capabilities  they  needed. 
Specifically  the  data  for  evaluation  was  available  to 
them  (and  to  external  software).  That  meant  that  the 
data  that  the  ITS  would  need  to  evaluate  the  BSG 
was  already  available  to  it.  Additionally,  there  were 
a  few  capabilities  required  by  the  ITS  that  proved  to 
be  beneficial  to  the  CAST  UP  system  as  a  whole  so 
that  they  were  designed  as  separate  modules  to  be 
used  either  by  the  ITS  or  by  other  CAST  UP 
components. 

After  the  design  for  each  module  was  complete  that 
module  was  implemented.  The  implementation  for 
this  project  was  different  than  most  past  ITS  efforts, 
but  it  is  becoming  more  typical.  Specifically,  the 
design  made  use  of  many  existing  components  which 
made  two  subtasks  more  prominent  than  usual. 
These  were  to  assemble  the  existing  components  and 
modify  them  as  required.  The  existing  components 
directly  utilized  by  the  ITS  effort  included  C2PC, 
OTB,  a  DIS  listener  and  emitter,  SimBionic,  and  the 
ITS  Authoring  Tool.  Modifications  included  add-ons 
to  C2PC  to  add  additional  editors  for  trainees  and 
instructors,  changes  to  OTB  affecting  how  it 
processed  some  specific  message  packets,  and  the 
DIS  listener/emitter  was  altered  to  capture  and  emit 
the  messages  required  by  the  ITS.  Components  were 
also  developed  and  interfaced  to  the  existing 
modules.  For  example,  the  software  to  check  for 
interference  of  simple  shapes  was  written  and 
integrated  into  the  ITS  evaluation  module. 
Additionally,  the  required  domain  knowledge  had  to 
be  entered  to  customize  the  generic  components  to 
the  BSG  domain.  This  included  defining  the  way  the 
evaluation  would  proceed  in  SimBionic,  defining  the 
principles  of  the  domain  (tactics,  techniques,  and 
procedures)  in  the  ITS  authoring  tool  and  attaching 
relevant  descriptions  to  these  principles,  and  defining 
attributes  of  the  scenarios  (such  as  what  principles 
each  tested  and  how  complex  each  was).  Of  course 
part  of  the  development  process  included  testing  the 
software  both  as  individual  components  and  in  an 
integrated  manner. 


Over  the  Summer  and  Fall  of  2003  the  BSGT  was 
evaluated  in  stages.  The  first  stage  was  an  internal 
evaluation  to  confirm  that  the  implemented  system 
met  the  client’s  and  the  CAST  UP  development 
group’s  expectations.  The  second  stage  was  an 
external  evaluation  through  demonstration  to  the 
instructors.  Their  comments  fed  back  into  the  design 
and  development  process.  In  the  third  stage,  the 
instructors  could  actually  sit  down  with  the  software 
and  play  the  roles  of  trainees  and  instructor.  We 
anticipate  the  fourth  stage  of  evaluation  will  be  a 
pilot  study  with  a  small  group  of  trainees  which  we 
expect  to  occur  early  in  2004.  Metrics  that  can  be 
used  to  determine  if  BSGT  is  meeting  the  training 
needs  described  earlier  include  the  amount  of  BSG 
scenario  practice  trainees  perform  before  they  arrive 
for  a  CAX,  the  amount  and  level  of  BSG  remediation 
required  from  human  instructors  during  the  CAX, 
and  the  level  of  training  that  can  be  accomplished 
during  the  CAX. 


RESULTS,  LESSONS  LEARNED,  FUTURE 
WORK 

The  overall  result  is  that  the  development  of  an 
Intelligent  Tutoring  System  that  can  provide  an 
intelligent  debrief  for  a  Combined  Arms  plan  is 
feasible  and  beneficial.  An  important  lesson  for 
others  developing  ITSs  is  that  the  standard  ISD 
process,  with  some  different  emphases  does  apply, 
especially  if  the  analysis  is  cognitive  and  involves 
both  the  tasks  to  be  trained  and  instructional 
decisions. 

There  were  several  lessons  relating  to  ITS  design. 
It's  important  to  get  the  ITS  developer  involved  in  the 
project  before  the  decisions  relating  to  them  have  to 
be  made,  if  at  all  possible.  Perhaps  the  most 
important  lesson  learned  was  that  an  architecture  for 
a  training  system  designed  to  maximally 
accommodate  the  instructors  will  also  accommodate 
an  ITS  with  little  or  no  instructional  compromises. 
An  additional  lesson  is  that  software  engineers  have  a 
tendency  to  want  to  include  in  the  design  of  editors 
used  by  trainees  helpful  features,  especially  if  they 
are  easy  to  implement.  However,  these  editors  must 
be  designed  to  allow  the  trainees  to  make  the 
mistakes  that  they  need  to  be  trained  to  avoid. 
Otherwise,  the  trainees  will  not  have  the  chance  to 
prove  that  they  would  avoid  those  mistakes  on  their 
own,  without  the  helpful  features.  This  is  especially 
important  if  these  helpful  features  will  not  exist  in 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2003 


actual  operations  (where,  for  example,  pencil  and 
paper  may  be  the  main  tools). 

Relating  to  implementation,  it  is  helpful  to  have  the 
developers  of  the  training  simulation  available  to 
make  slight  modifications  to  accommodate  the  ITS's 
requirements.  A  current  DARPA  sponsored  effort, 
DARWARS,  among  other  things,  is  seeking  to 
develop  simulation-ITS  interoperability  standards 
which  should  alleviate  this  need.  During 
development  some  of  the  ITS  required  software 
capabilities  may  be  more  generally  useful  and  should 
be  modularized  and  offered  for  use  by  the  other 
components. 

There  are  many  facets  to  the  planned  future  work. 
Most  immediately,  the  BSGT  will  be  further 
developed,  evaluated  with  trainees,  and  fielded. 
Specifically  more  effort  will  be  placed  on  increasing 
the  breadth  of  the  evaluation  of  the  FiST  developed 
CA  plan  and  tutoring  the  individual  and  team 
performance  of  the  FiST.  The  execution  phase  of 
CA  FiST  training  will  be  addressed  beyond  the 
existing  simple  conflict  checking  capability.  CAST 
UP  serves  a  number  of  other  team  members,  many  of 
whom  are  also  slated  for  the  development  of  an  ITS. 
Current  plans  call  for  efforts  targeting  members  of 
the  aviation  community  involved  in  Combined  Arms 
such  as  AH-1W,  F/A-18,  and  AV-8B  crew, 
intelligence  specialties,  etc. 

Additionally  beyond  these  ITS  for  individuals,  long¬ 
term  plans  include  the  development  of  ITS  targeting 
the  team  as  a  whole  as  well  as  teaching  principles  of 
teamwork  within  that  team.  Over  time,  additional 
capabilities  will  be  developed.  To  teach  teamwork 
an  ITS  will  need  to  understand  the  verbal 


communication  that  occurs  between  team  members. 
At  least  in  some  circumstances,  these 
communications  are  highly  structured  and  rooted 
deeply  in  the  tactical  context.  Developing 
capabilities  for  these  kind  of  communications,  while 
challenging,  is  still  tractable. 

Another  capability  that  we  have  begun  to  develop  for 
other  tactical  tutoring  applications  and  that  may 
eventually  be  added  to  the  CAST  UP  ITSs  is  the  use 
of  the  Socratic  method  in  After  Action  Reviews 
(AARs).  The  Socratic  AAR  component  uses  the 
same  information  that  the  simple  AAR  component 
presents  to  the  trainees  but  instead  of  merely  telling 
the  trainees  what  decisions  were  right  and  wrong  and 
why,  it  asks  the  trainees  a  series  of  questions,  with 
the  goal  of  getting  the  trainees  to  determine  for 
themselves  what  the  correct  decision  was  and  why. 
Similarly  to  hinting,  the  goal  is  to  use  the  most 
general  questions  first  and  only  proceed  to  more 
specific  ones  if  the  trainees  are  having  difficulty 
determining  the  cause  of  the  mistake. 
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